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ABSTRACT 

A plan for evaluating the Apple Classroom of Tomorrow 
(ACOT) is presented, which implements the model for Sensible 
Technology Assessment/Research (STAR) • STAR involves interactive 
participation in the evaluation study by ACOT participants, 
collateral university researchers, and University of California (Los 
Angeles) staff to develop a credible adaptive set of assessment 
plans, procedures, and reports assessing the ACOT experiment; a 
phased implementation of STAR designed to conform to the rhythms of 
site-by-site implementation; and a focus on exploring the utility of 
developing STAR into a multi-user evaluation system for future 
coordinated implementation with new technology adoption in local 
districts. The STAR evaluation will be guided by questions 
concerning: (1) the effects of ACOT on students; (2) the influence of 
ACOT on organization and delivery of instruction; (3) the effects of 
ACOT on teachers; (4) the effects of ACOT on students* families; and 
(5) unintended effects that may be attributed to ACOT. The 
comparative impact of ACOT will be investigated for three groups of 
students: ACOT students over time; students at one grade level taught 
by an increasingly experienced ACOT teacher; amd ACOT and non-ACOT 
students in the same and different schools. Investigation of the 
effects of ACOT on students will consider academic achievement, 
criterion-referenced student writing, problem solving, locus of 
control or sense of efficacy, attitudes toward school or motivation 
for schooling, academic self -concept, future educational and career 
plans, use of time at home^ and site-specific and instructional 
goals. Similar issues will be considered for teachers and families. 
The evaluation was scheduled to begin in Spring 1988 at three 
seTected ACOT sites in Memphis (Tennessee), Nashville (Tennessee), 
and Columbus (Ohio) . (SLD) 
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Introduction 



This document presents a plan for evaluating the Apple Classroom of Tomorrow 
(ACOT) Our plan implements the model for Sensible Technology Assessment/Research 
(STAR) and has multiple antecedents. First, it derives from our historical perspective of 
critical issues and approaches to the assessment of innovation. More specif cally, it 
embodies principles identified In the paper submitted to Apple Computer in December, 
entitled "Sensible, Sensitive Technology Assessment. " More '^mediately, it has a 
derived, as well as Interactive relationship with the flrst-cut model of STAR, briefly 
pictured and described on a companion document, "A Model for Sensible Technology 
Assessment/Research (STAR)." It is our hope to adapt the STAR model and 
Implementation mutually and dynamically as we progress through various phases of the 
ACOT evaluation. 

Among the major points of STAR are the following: 

• Interactive participation in the evaluation study by ACOT participants, 
collateral university researchers, and UCLA staff to develop a credible, 
adaptive set of assessment plans, procedures, and reports assessing the ACOT 
experiments. 

• A phased implementation of STAR designed to conform to the rhythms of 
site by site implementation 

• A focus on exploring the utility of developing STAR into a multiuser 
evaluation system, for future, coordinate implementation with new 
technology adoption in local districts 

The STAR approach is at once simple and complex, depending upon one's view. 
It relies upon our Implementing a model of evaluation, cycling through differential 
emphases, as appropriate to decision requirements, the model and available resources, 
and determining which questlons-procedure-analysis configurations are useful, credible, 
and technically sensitive enough to deserve a major slot in the optimal STAR system. 
The evaluation will be guided by a relatively straightforward set of questions: 

• What are the effects of ACOT on students? 

• How does ACOT influence the organization and delivery of instruction? 

• How does ACOT affect teachers? 

• Wh3t are ACOT effects on the family of ACOT students? 

• What other unintended effects, either positive or negative, may be 
attributed to ACOT? 



Technical Approach 

The ways we find answers to these Issues relates to two major decision arenas: 
the design for the study and the measures used to obtain evidence on each of the 
questions. 

Design Overview 

In the broadest sense, our design will start small and simple and grow In 
complexity, requirements, and power over a three-to- five year period. We believe that 
we will be collecting, in large part, baseline information during 1988. We are interested 



In one primary target during the first phase of work: students outcomes. Althougl. 
students data will be our emphasis, we will attend also to questions related to teach s 
and classroom process, perhaps using already in place ACOT research components. Our 
own focus, and whether to augment or supplant these approaches will be reserved for 
later years. We will also extend our interests to family and to school level processes and 
effects during subsequent phases of the study. 

Student Targets 

Specifically, we are interested in knowing the comparative impact of ACOT on 
three cohorts of students: ACOT students over time as they go up the grade structure of 
schools; students taught subsequently by an inaeasingly experienced ACOT teacher 
team, for instance 5th grade students in 1988 compared to Sth grade students in 1989, 
and on ACOT compared to non-ACOT students in their same and different schools. 

The designs for such studies will take the following form, illustrated in Figure 1, 
in order to facilitate the following comparisons of three groups. 



Figure 1 



Design for Student Outcomes 



Minimum 

Measurement 

Schedule 


ACOT Students 


Non*ACOT Students 
in ACOT schools 


Non-ACOT Students 
in non-ACOT schools 


Time 1 
Spring 1966 








Time 2 
Fall 1969 








Time 3 
Spring 1969 
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In implementing this design, we would dearly expect to have greater data 
collection intensity for ACOT participants. We need to assure that comparisons 
between ACOT and non ACOT students can be made longitudinally both for classrooms 
and Individuals so that we can capture both the cumulative and delayed effects of the 
experience that are undetectable Immediately. 

Secondly, and to a lesser degree at the outset, we will be Interested in effects on 
teachers and would hope to use a similar but less extensive design In collecting teacher 
data. Our approach for the first year will be to piggyback on other classroom 
observation measures and to administer relatively brief instruments to teachers to elicit 



their perception of ACOT effects on classroom process, worklife, and students. The 
teacher focus is to be defened not because of any absolute value, but because of our 
sense of teachers' heavy existing workload and data requirements in the project. Should 
teachers' become more available, even this year, we will be ready to create additional 
measures and procedures as desired. 

Similarly, we expect to be relatively light in our efforts to develop new 
approaches to the assessment of instructional organization during this first year (See 
Levine's attached plan as a partial ide-*, set), partly because we are aware of the parallel 
efforts serving these issues, underta'^en by the ACOT project. However, some of our 
attention to teachers and classroom process will be focused in this and subsequent years 
on the issue of identification of new objectives and new outcomes that can be 
subsequently verified, validated, and incorporated as legitimate ACOT outcomes, 
optional ACOT objectives for participants, and certainly, as measures in the STAR plan. 



Core Measures in STAR 

Below Is a set of measures we believe useful in addressing the evaluation 
questions we have listed. In conformance with our model, we will be trying to 
determine from all principal players, first, If there are other questions which should be 
addressed, and second. If the measures we have suggested are appropriate. 

What Are the Effects of ACOT on Students? 

Interviews with ACOT teachers, coordinators, and administrators as well as on-site 
observation suggest a common core of outcomes which are of Interest and potentially 
applicable across the sites. 

Student achievement in basic academic skills. We believe we need to 
develop alternative approaches that attempt to optimize both cross-site comparisons 
and within site sensitivity. Because of district mandates, each of the ACOT schools 
routinely administers standardized tests of reading, mathematics and language, and 
scores on these measures over time are easily available through cumulative folders for 
ACOT and non-ACOT students. 

We also believe it essential to have comparable standardized outcome data aaoss 
sites. Because of grade level, community and currlcular differences, finding a basis for 
comparison is always difficult. It is the apples and oranges problem, once again, no pun 
Intended. Nonetheless, we have considered a number of options at this point. One 
would involve employing a linking and equating (or anchor test) approach to attempt 
to convert disparate district achievement measures to the same scale. This approach 
requires a sll^tly larger sample size and is most useful for comparisons across sites 
operating at the same grade level. 

Another choice is to make comparisons where natural matches occur. This year, 
for instance, the Memphis and Nashville sites use the same standardized measure in 
their ACOT classrooms. However, because district test policies may change at different 
grade levels or change altogether, relying on fortuitous matches seems to be risky In the 
long run. 

A third option, and one we endorse, Involves using a test battery that Is 
vertically equated over the broad age ranges of potential ACOT implementation. We 
believe that the Iowa Test of Basic Skills, or at least some subscales thereof, is an 
appropriate measure. The ITBS has relatively strong vertical equating (increases are 
comparable from grade to grade) and uses a uniform norming procedure. Thus, we would 
be able to compare aaoss sites the progress students make using percentile 
performance, even for very different grade levels and subjects. We have greater 



technical explanations for this preference, which can be presented, if desired. We also 
understand this option requires greater local site participation and negotiation. In any 
case, we do not subsaibe to the proposition that such a general test as the ITBS will be 
sufficiently sensitive to local site emphases. However, local sensitivity could be 
increased by a procedure where teachers and curriculum experts matched the ITBS 
scales with areas of local currlcular and instructional emphasis. As an added benefit, 
certain ITBS subscales could also be used as the anchor test items, serve to link disparate 
local site measures, and allow us to incorporate our first option as well. 

Fourth we may be able to capitalize in CSE's participation In other national 
testing activities to provide comparable, standard measures. For Instance, all 
participating sites are located In states with state assessment programs. As part of the 
Test Assessment Center of the Council of Chief State School Officers, a cooperative 
project developing common measures for school subject has been undertaken. Access 
and approval for use of this measures Is possible. Clearly the selection of any approach 
will need to Involve the site participation and agreement. 

Criterion-referented student writing. All teach;}rs commented on 
improvements In the quality and quantity of student writing resulting from the ACOT 
experience, changes which will be explored using CSE developed and validated analytic 
scoring schemes. The schemes provide diagnostic as well as summary Information about 
the quality of students' writing performance and may enable schools to analyze the 
strengths and weaknesses of their writing instruction and of Individual students. The 
schemes were the basis for the lEA (International) study of students writing as well as for 
a number of state and local assessments. Not only do well defined and reliable 
procedures for training scorers (teachers and/or curriculum stafO and scoring student 
essays exist, but use of the prompts (assignments) from from the international study will 
enable comparisons of ACOT student performance with national and International 
samples. These prompts Involve typical written composition genre, narration, 
exposition, and desalptlon. 

The writing assessment will be used not only to assess the quality student writing, 
but also to gather additional data on the nature of ACOT effects. That is, we will ask sets 
of some students within each classroom to write about the effects of their ACOT 
experience, about their school year, or about their plans or expectations for the future. 
By conducting content analyses of responses, over time and between ACOT and non- 
ACOT students, we will have an interesting and flexible method to examine In depth a 
variety of outcomes. C:SE, with federal support, is presently conducting research and 
adapting the scoring schemes for use to assess deep content knowledge. If our efforts 
are successful, these content-focused measures could also be employed In selected ACOT 
sites. ACOT and comparable non-ACOT students will be asked to write essays at the 
beginning, middle, and end of the school year, using matrix sampling of prompts within 
each classroom, as indicated in Figure 1. 

Problem-solving. While teachers and coordinators each spoke about changes 
students problem solving, they had difficulty In operaionallzing the nature of the 
changes, making the choice of specific Instrument premature. During Phase I of the 
evaluation, we will be further Investigating the appropriateness of available 
problemsolving measures while we gather additional qualitative data about the nature of 
apparent changes (see appendix for description of qualitative approach). We shall be 
looking for both cognitive and affective components In this area (i.e., measures of how 
student solve problems and potential problem solving consequences of work with 
sprecd sheets, databases, etc). We will also attempt to assess their willingness approach 
complex problems and to troubleshoot and explore alternative solutions before asking 
for help. Further, one might expect the database and spread sheet work to influence 
analysis skills. It may well be that different ACOT sites influence different kinds of 
problem-solving skills and will require different types of measures for the various sites. 
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We anticipate Identification of appropriate problem-solving measures and their 
preliminary tryout will consume Phase I activity In this area. Phase II would see 
administration and analysis of measures in ACOT and comparable non-ACOT classrooms. 

Locus of control or sense of efficacy. The sense of empowerment and 
efficacy students gained from their experiences with computers was stressed across sites. 
Because the locus of control construct and attribution theory has enjoyed a fair amount 
of theoretical attention, a number of measures of this construct exist— however, with 
varying credibility and technical underpinnings. CSE used the Intellectual Achievement 
Responsibility Scale (lAR, a research instrument) In an early study of state-wide reform 
in California, and can provide some point of comparison for ACOT, albeit dated. The 
commercially published School Attitude Measure (SAM) Includes an Instructional 
Mastery subscale, another source for this area. The SAM instrument has the advantage 
of being nationally normed, while the lAR derives from a stronger research base. After 
verifying that these two measures are the best of their type, we would plan to 
administer both Instruments and then to use the results from Phase I to conduct validity 
analyses. 

Attitudes toward school or motivation for schooling. Like the locus of control 
area, a number of both commercially published and public domain measures are 
available, Including those CSE has used in other studies. We propose a parallel approach 
to the locus of control Issue (I.e., administer the SAM and a shorter, public domain scale 
during Phase I and use validity studies from these data to verify these selections for 
Phase II data collection). 

Student attendance data, gathered from archival data (I.e., cumulative folders) for 
ACOT and comparable non-ACOT students will provide additional evidence of attitudes 
toward school and motivation for schooling. 

Academic self-concept. Data will be gathered using the SAM, enabling 
comparisons of ACOT students to a national nonn group and to themselves over time 
(prior to, during and subsequent to ACOT). 

Future educational and career plans. The plans for writing assessment 
summarized above mentioned plans to analyze the content of student essays on this 
topic. Questionnaire items also will be developed, particularly to assess changes over 
time. 

Use of time at home. The amount of TV watching and book reading will be 
investigated in student and parent questionnaires. Ultimately, on-line records of 
computer use for homework, paid work, or leisure by ACOT students would be desirable. 

Site-specific curricular and instructional goals. Phase I of the evaluation will 
be used to identify important, site-specific goals. These will be a target of assessment 
during Phase II. 

How Does the ACOT Approach Influence the Organization and Delivery of 
Instruction? 

The following will be investigated through a combination of classroom 
observation and student and teacher questionnaires (Additional detail on qualitative 
observation approaches can be found in appendix). The distribution of effort by CSE 
against these Issues will be limited in year one, with the exception of teaching and 
instructional goals. 

I. Teaching and instructional goals 



2. Degree of individualizatlon/group work 



3. Currlcular balance (Ihe concern that computer literacy and keyboardlng skill 
development may be pushing out science and social studies; availability of 
software in these latter subjects also push in this direction) 

4. Pacing 

5. Nature of student/teacher dialogue 

6. Nature and frequency of teacher-student interactions 

7. Nature of student-student interactions/amount of peer tutoring. 

8. Time on task 

9. Amount and completion of homework 

10. Nature of instructional planning 

We understand collateral ACOT efforts are directed to many of these issues. It is 
our interest to explore to the extent to which some of these measures can be collected 
and aggregated in automated form. 

How Does ACOT Affect Teachers? 

The following teacher issues could be examined via classroom observation, 
teacher and administrator questionnaires and detailed interviews. CSE project attention 
on this general area, however, cannot reasonably begin until we get a sense that 
teachers can handle the data requirement. 

1. Role (e.g., manager vs. lecturer) 

2. Sense of professionalism 

3. Efficacy 

4. Enthusiasm/burnout 

5. Expectations for students 

6. Leadership/professional development 

7. Comfort, ownership of technology 

8. Use of time/total time 

What are ACOT Effects on the Family of ACOT Students? 

A brief parent survey and documentation of attendance at school events, of 
help, and of volunteering will be prepared. Specific issues will include: 

1. Parental attitudes toward school 

2. Parental involvement/willingness to be involved in school activities 

3. Parental expectations for their children 

4. Parental perceptions of ACOT effects, changes in their children 
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5. Changes In parental exp.ctatlons for themselves or other family members 



What Other Unintended Effects, Positive or Negative, May Be Attributed to ACOT? 

Hypotheses are developed both by the analysis of explicit questions posed to 
participants and in the explanations posited for relationships among data found in the 
study. Below is a model, Figure 3. that simply illustrates the utility of this information. 
However, the search for unanticipated effects is as much a frame of mind as a set of 
procedures. 



Procedures and Timelines 

The evaluation plan, as alluded to above, anticipates a multiphase effort. Phase I 
will concentrate on improving the model, testing the feasibility of the design, and 
refining data collection approaches. Included in this first phase (Spring, 1988) will be 
collection of baseline data In prespecified and agreed areas (student academic 
achievement, writing, locus of control, attitudes toward self and school). We will also 
undertake additional exploratory work to permit specification and selection of 
appropriate instrumentation (problemsolving, classroom process), and additional 
information gathering to identify site specific curriculum and instructional goals which 
should be assessed. This work is planned to be conducted in the three selected ACOT 
sites, Memphis, Nashville, and Columbus. 

Phase I! and beyond will expand the scope to all ACOT sites, with an iterative 
process of collaborative planning, data collection, analysis, reporting, and identification 
of refined goals. It will also extend the intensity and breadth of measure, or reduce 
them, as appropriate. We will begin the collection and analyses of data on all core 
outcome measure and aoss-sectional comparisons of ACOT and comparable non-ACOT 
students. Simultaneously, we will be engaged in database design and implementation. 
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